Intelligent background selection and image segmentation

ABSTRACT

Certain embodiments of these teachings take inputs of a) an image to be segmented and b) a set of user input markers that identify at least a foreground of said image. From these inputs are determined strong foreground regions S f , strong background regions S b  and ambiguous regions G. Each ambiguous region G i  is classified as either a weak foreground region W f  or a weak background region W b  by comparing color of each ambiguous region G i  against color of regions adjacent thereto and merging the respective ambiguous region G i  with its adjacent region having a most similar color. The strong foreground regions S f  are also merged with the weak foreground regions W f . What is output therefrom is a foreground image of the image to be segmented, separate and distinct from any region that is classified as one of the strong background regions or weak background regions.

TECHNICAL FIELD

The invention relates generally to multimedia applications; image andvideo editing; image cut-outs; and selective filtering. Specifically,embodiments of the present invention relate to methods for imagesegmentation to separate foreground from background.

BACKGROUND

Image segmentation is a challenging problem in computer graphics,particularly with respect to digitally separating a desired object(foreground) from a background of the digital image. Such separation isuseful during digital editing of still images and video. Fully automaticsegmentation has been a very difficult task to effectively apply over awide variety of images due to the difficulty in segregating a desiredobject from a background in an image. For this reason, methods forinteractive image segmentation have recently received much attention.

Several approaches have been suggested, and some of which are relevantto these teachings are described in the following documents. [1] FASTINTERACTIVE IMAGE SEGMENTATION BY DISCRIMINATIVE CLUSTERING, Proceedingsof the 2010 ACM Multimedia Workshop on Mobile Cloud Media Computing,MCMC '10, pages 47-52, 2010, by Dingding Liu, Kari Pulli, Linda G.Shapiro and Yingen Xiong; [2] INTERACTIVE IMAGE SEGMENTATION BY MAXIMALSIMILARITY BASED REGION MERGING , Pattern Recogn., 43(2): 445-456,February 2010, by Jifeng Ning, Lei Zhang, David Zhang, and Chengke Wu;[3]“GRABCUT”: INTERACTIVE FOREGROUND EXTRACTION USING ITERATED GRAPHCUTS , ACM SIGGRAPH 2004 Papers, SIGGRAPH '04, pages 309-314, 2004, byCarsten Rother, Vladimir Kolmogorov, and Andrew Blake; [4] A MULTILEVELBANDED GRAPH CUTS METHOD FOR FAST IMAGE SEGMENTATION, Proceedings of theTenth IEEE international Conference on Computer Vision (ICCV '05), ICCV'05, pages 259-265, 2005, by Hez-ve Lombaert. Yiyong Sun, Leo Grady, andChenyang Xu; [5] LAZY SNAPPING , ACM SIGGRAPH 2004 Papers, SIGGRAPH '04,pages 303-308, 2004 by Yin Li, Jian Sun, Chi-Keung Tang, and Heung-YeungShum; and [6] CONTOUR DETECTION AND HIERARCHICAL IMAGE SEGMENTATION ,IEEE Trans. Pattern Anal. Mach. Intell., 33(5)898-916, May 2011, byPablo Arbelaez, Michael Maire, Chanless Fowlkes, and Jitendra Malik.

These approaches generally require significant user guidance and areparticularly inaccurate when an object is close to a boundary of theimage. At least references [1-5] require the user to manually mark botha background region and foreground region. The inventors have found thatthese approaches tend to be less accurate in discriminating foregroundfrom background when the true foreground extends to or very near aborder/outer boundary of the overall image being segmented. Thereforethere is a need for an image segmentation approach that betterdistinguishes foreground from background at least in those instances,and preferably one that does so with fewer manual user inputs so as toresult in a generally more automated system.

SUMMARY

Embodiments of the present invention are directed to an apparatus suchas a host computing device, a method and a computer readable memory thatsatisfy the need for reducing user guidance for background selection andimage segmentation while maintaining accurate results in discriminatingforeground from background.

A method having certain features of these teachings comprises:

-   -   using a set of user input foreground markers identifying a        foreground of an image to be segmented, defining N_(f)        foreground regions and N_(b) background regions, where N_(f) is        an integer greater than zero and N_(b) is an integer at least        zero;    -   for each respective border region that defines a border of the        image and is not one of the N_(f) foreground regions, performing        a multi-level assessment of the respective border region to        determine whether the respective border region is a possible        foreground region, where each level of the multi-level        assessment compares color similarity, with respect to a        threshold that is specific for that level, between the        respective border region and any of the N_(f) foreground regions        or of a region that satisfied the threshold of a previous level        of the multi-level assessment;    -   from the multi-level assessment and the foreground markers,        classifying each region of the image to be segmented as one of a        strong foreground region, a strong background region or an        ambiguous region;    -   classifying each of the respective ambiguous regions as a weak        foreground region or a weak background region; and    -   merging the strong foreground regions with the weak foreground        regions and output therefrom, to a tangible memory or to a        graphical display, a foreground image of the image to be        segmented, separate and distinct from any region classified as        one of the strong background regions or weak background regions.

Another method according to these teachings comprises:

-   -   from inputs of an image to be segmented and a set of user input        markers identifying at least a foreground of said image,        determining strong foreground regions S_(f), strong background        regions S_(b) and ambiguous regions G;    -   classifying each ambiguous region G_(i) as either a weak        foreground region W_(f) or a weak background region W_(b) by        comparing color of each ambiguous region G_(i) against color of        regions adjacent thereto and merging the respective ambiguous        region G_(i) with its adjacent region having a most similar        color; and    -   merging the strong foreground regions S_(f) with the weak        foreground regions W_(f) and outputting therefrom, to a tangible        memory or to a graphical display, a foreground image of the        image to be segmented, separate and distinct from any region        classified as one of the strong background regions or weak        background regions.

A computing device or other such apparatus embodying the teachingsherein may comprise at least one digital processor and a computerreadable memory storing a computer readable program/computer executablecode which when executed by the at least one processor causes theapparatus to perform either or both of the methods essentially assummarized above. Other embodiments implementing these teachings maycomprise the computer readable memory on which is stored such a programor executable code, which may then be distributed to end users who maythen install the program from that memory onto their own computer'snative memory or have their computer run the program directly from thenon-native memory itself.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentinvention will become better understood with regard to the followingdescription, appended claims, and accompanying drawings, where:

FIG. 1 shows a high level flow chart according to an example embodimentof the invention.

FIGS. 2A and 2B show an image illustrating a user-marked foregroundregion and also showing border regions that are by default are at firstconsidered background regions according to various embodiments of theinvention.

FIGS. 3A and 3B illustrate various types of images to be segmented (leftcolumn) and the foreground portions that are segmented and isolatedtherefrom (right column) and which are output by exemplary embodimentsof the invention.

FIG. 4 is a further image to be segmented having user markersidentifying foreground regions, and showing various other regions of theimage that are evaluated by an algorithm according to these teachings asto whether they are foreground or background.

FIG. 5 is a process flow diagram showing operation of an apparatusaccording to certain exemplary embodiments of these teachings.

FIG. 6 is a high level schematic block diagram illustrating a computingdevice that is suitable for practicing the described and exemplaryembodiments of these teachings.

DETAILED DESCRIPTION

In the Summary above and in the Detailed Description and the claimsbelow, and in the accompanying drawings, reference is made to particularfeatures (including method steps) of the invention. It is to beunderstood that these various features may be combined despite that thedescription herein does not explore explicitly every single suchpossible combination. The specific embodiments that are detailed withparticularity herein are not a limit to the general teachings herein,and such exemplary embodiments by themselves are not a limit to thescope of the ensuing claims but rather teach the principles of theinvention and illuminate the meaning of such ensuing claims.

The documents mentioned in the background section describe interactiveimage segmentation approaches. An interactive image segmentationapproach requires some user inputs, typically markers created and inputby a user that show which portions of an image are background portionsand which portions of the image are foreground portions. If bothforeground and background have the same shades of color, then analgorithm must decide whether to omit the color which is similar toforeground when segmenting the image. Background segregation from anassumed default background plays an important role in accuratesegmentation. The default background is obtained from the region ofinterest (ROI) provided from the user.

Efficient energy minimization algorithms, such as graph cuts, are widelyused to label the unmarked pixels in interactive segmentation, forexample by documents [4] and [5] in the background section. The mainsteps from document [5] are object marking followed by boundary editing.However, research from document [2] confirmed that accurate segmentationof an image requires enough foreground and background samples. Document[2] suggests using a color histogram of a region to perform MaximallySimilar Region Merging (MSRM) by which to merge small regions intobackground. The red/green/blue (RGB) colors are uniformly quantized into16 levels per channel and histograms of 16×16×16=4096 bins are used toestimate region similarity. Although this resulted in bettersegmentation quality than if graph cuts were used with the same markers,document [2] considers only local information in this merging process.Each of the approaches described by documents [1-5] requires extensiveuse of background and foreground markers and often do not perform propersegmentation when the object is at the border/boundary of the imagebeing segmented.

Embodiments of the invention address these limitations by providing (i)robust object segmentation with intelligent background selection withrespect to user marked foreground; (ii) refinement in usage of theobtained foreground regions; and (iii) object segmentation using veryfew or even no background markers.

FIG. 1 shows a high level flow chart of embodiments of the invention.Each step of the flow chart is detailed more thoroughly below. The setof foreground markers, and optionally also background markers, are thefirst inputs to be provided from the user at step 100. Low-levelpre-segmentation is done at step 102, for example using a mean-shiftanalysis technique. The pre-segmentation is followed by automaticbackground selection at step 104 which is performed based on a defaultbackground. Using these inputs, discriminating filtering is applied forall regions of the image and local neighborhood classification usingstrong and weak foreground and background information. These arerepresented in FIG. 1 as steps 106 and 108. The image is pruned at 110resulting in the segmented foreground object at step 112. The entireprocess will reduce the number of foreground markers required andrequire few, if any, background markers. This is the case even when aforeground portion extends to the border of the image being segmented.

Typically for image and video editing the segmented foreground objectresulting at step 112 is output, separate and distinct from theremaining background object of the original image that has beensegmented. In some cases the outputs will be both foreground andbackground objects as separate and independent digital files. In anycase each output is a digital file, which may be displayed on-screensuch as when the digital file remains stored within the internal memoryof the computer which performed the image segmentation, or the outputmay be a removable memory on which is stored such digital file(s), orthe output may be a transmission of the digital file(s) over a wired orwireless communication pathway for storage on an external memory.

In some embodiments of the invention the only user markings on the imagethat is input for segmentation are of foreground areas, while in otherembodiments it is advantageous to additionally have user markings ofcertain background areas. In the prior art referenced in the backgroundsection above user markings are required for both foreground andbackground areas. Assume a user has marked the desired features of animage indicating only a foreground object. These are the user inputs atstep 100 of FIG. 1. At low-level pre-segmentation 102, the image isfirst over-segmented using for example a mean shift algorithm. The meanshift algorithm is a suitable example as it preserves boundaries well inmost cases and provides less over segmentation as compared to other lowlevel image processing techniques such as a watershed algorithm, butthis is only one non-limiting technique for the pre-segmentation step102. However implemented, the pre-segmentation step 102 uniquely labelseach region, denoted by L.

At automatic background selection step 104, based on the set of inputsmarkers by the user, a number N_(f) of foreground regions M_(f) aredefined, and optionally, a number N_(b) of background regions M_(b) aredefined. Necessarily the number N_(f) of foreground regions is at least1 since the user has input a marker identifying it as such, and while inmost implementations there will be background regions identified it ispossible that, without a user marked background region the number N_(b)of background regions may in fact be zero at this stage. A defaultbackground region B is selected as ROI border regions. In oneimplementation all border regions of the image that are not marked bythe user as a foreground region are considered initially as such defaultbackground regions. In exemplary embodiments of the invention, thedefault background regions are scrutinized against the user markedforeground regions according to the following steps:

-   -   (1) For each region {B_(i)}_(i=1,2, . . . m), a set of adjacent        regions is formed, denoted as S _(B)={A_(j)}_(j=1,2, . . . n)        for n adjacent regions. An inter color distance d_(j) is        calculated using mean colors in CIELab color space between each        region of B_(i) and its adjacent regions S _(B). CIE is a term        of art in the image processing arts representing a uniform color        space or scale in terms of RGB, and is not limiting to the        broader teachings herein. A Minimum of (d_(j)) is computed to        find the most similar adjacent region, which is denoted below as        the k^(th) region. The regions are merged under the following        criteria:

if S _(k) ^(A) ^(j) ε{B _(i)} then, S _(k) ^(A) ^(j) ∪B _(i)

If the above criteria is not met then S _(k) ^(A) ^(j) and B_(i) willnot be merged. New labels are generated L_(new) and new border regions,B_(new) are obtained. This step allows similar neighbor border regionsto be merged into a single region which provides savings to computationcost during execution of the remainder of the algorithm that is detailedbelow.

-   -   (2) Each distance of {B_(i) ^(new)}i=1,2, . . . m regions is        scrutinized against the distance of M_(f) user marked foreground        regions and the minimum color distance differences restricted        with a threshold thr are found by the following equation:

diff D _(i)=min(d(M _(f))−d(B _(i) ^(new) _(i=1,2, . . . m)))

-   -   This step determines the border regions that are similar to the        user marked foreground regions M_(f).

(3) The regions satisfied by step (2) immediately above are denotedbelow as B_(sim). With reference to B_(sim), a color histogram iscomputed for each border region. In exemplary embodiments, the RGB colorspace is used to compute the color histogram. Each color channel is, forexample, uniformly quantized into 8 levels and a histogram of eachborder region is calculated in the feature space, for example, of8×8×8=512 bins. The normalized histogram of each border region B_(new)is denoted by Hist_(B) _(new) . The Bhattacharya coefficient, ρ, is usedto measure the similarity between m^(th) and n^(th) border regions,

${\rho \left( {m,n} \right)} = {\sum\limits_{u = 1}^{512}\; \sqrt{{Hist}_{m}^{u} \cdot {Hist}_{n}^{u}}}$

where ρ varies from 0 to 1 and superscript u represents the u^(th)element of normalized histogram bin. As is known in the image processingarts, the higher the Bhattacharya coefficient between m and n, thehigher the similarity is between them. The normalized histogram Hist_(B)_(new) for each border region is computed and the similarity is measuredonly for the normalized histogram Hist_(B) _(sim) border regions in thisembodiment. The similar regions, designated B_(mat), are finallyselected with the following criteria:

ρ(Hist_(B) _(new) ,Hist_(B) _(sim) )>X

where X is the minimum threshold set for similarity criteria. This stepdetermines the spatially disconnected similar border regions withrespect to B_(sim), but also finds similarity for texture relatedregions.

Using this information, the possible foreground and background regionsare obtained.

Possible foreground regions P_(f) are obtained using:

P _(f) =B _(mat) ∪B _(sim).

and, possible background regions P_(b), are obtained using:

P _(b) =B _(new)

P _(f).

The possible background regions, B, obtained by step 104 are depicted inFIG. 2A as all of the border regions towards the edges of the exampleimage, since in the FIG. 2A-B example no border region is marked by theuser as being foreground. The star 200 represents a foreground object.Initially, all of the border regions not marked as foreground are set toa default background based on steps 100 and 102. Regions 202 and 204represent default background regions. Automatic background selection 104determines the possible foreground regions P_(f) which are representedby region 204. Now referring to FIG. 2B, the possible foreground regions206 have been removed from the background regions which results in a newset of possible background regions, namely P_(b). That is, at FIG. 2Bthe shaded border regions are those which are still possible backgroundregions P_(b), whereas the unshaded border regions 206 are those whichhave been determined via a color matching analysis to be sufficientlyclose to a user designated foreground region 200 that they are nowconsidered possible foreground regions P_(f).

In exemplary embodiments of the invention, at step 106 a set of regionsB_(init) is created by P_(b) U M_(b), where M_(b) represents the usermarked background, if any. If the user has not marked any backgroundregions, then M_(b) is assigned as NULL. At least B_(init) and the usermarked foreground M_(f) are used as inputs for implementing adiscrimination filter. Reference [1] in the background section abovedetails discrimination filters and so such filters need not be furtherdescribed herein. One particular such discrimination filter uses a k-dtree to classify foreground and background regions, and this is the typeof discrimination filter used in the example algorithm according tothese teachings which generated the isolated foregrounds shown at FIGS.3A-B. However implemented, the discrimination filter is configured tooutput the following three types of regions: strong foreground regionsS_(f); strong background regions S_(b); and ambiguous regions G.

After classifying the strong foreground regions S_(f) and strongbackground regions S_(b), the ambiguous regions G are then classified aseither weak foreground W_(f) or weak background W_(b) regions at step108. To accomplish this, local neighbor region information is usedaccording to the following example process:

-   -   (i) For each ambiguous region {G_(i)}_(i=1,2, . . . m), a set of        adjacent regions S _(B)={A_(j)}_(j=1,2, . . . n) is formed. The        mean colors in the CIELab color space are used to calculate the        inter color distance between each i^(th) ambiguous region G_(i)        and its adjacent regions S _(B). In principle this is similar to        the process described above regarding step 104. The minimum        color distance (d_(j)) is computed to find the most similar        region (k^(th) region). The ambiguous regions G_(i) are then        labeled as

${L_{new}\left( G_{i} \right)} = \left\{ \begin{matrix}W_{f} & {{{if}\mspace{14mu} {\overset{\_}{S}}_{k}^{A_{j}}} \in \left\{ S_{f} \right\}} \\W_{b} & {{{if}\mspace{14mu} {\overset{\_}{S}}_{k}^{A_{j}}} \in \left\{ S_{b} \right\}}\end{matrix} \right.$

If S _(k) ^(A) ^(j) ε{G_(i)}, then merge:

S _(k) ^(A) ^(j) ∪{G _(i)}

and repeat step (i).

This merging process classifies each ambiguous region as either a weakforeground region W_(f) or a weak background region W_(b) to satisfystep 108 of the overall process at FIG. 1. In one embodiment neither theforeground nor background list is updated yet with these classified weakregions to avoid unwanted labeling and user confusion.

In the pruning step 110, some or all of the following three steps areperformed to obtain the segmented foreground object 112:

-   -   1) Merge the strong and weak foreground regions SW_(f):

SW _(f) =S _(f) ∪W _(f)

-   -   Similarly, merge the strong and weak background regions SW_(b):

SW _(b) =S _(b) ∪W _(b);

-   -   2) Classify any gap of foreground segmentation SW_(f) which is        surrounded by foreground region as foreground region; and    -   3) Remove isolated foreground regions depending upon the user        foreground marker.

Referring now to FIG. 3A, a first column 300 and a second column 302 ofimages are shown. The images shown in the first column 300 represent awide range of image types that are first input for segmentation in theinventors' testing of a specific embodiment of these teachings.Specifically, the gorilla image, in the first row of FIG. 3A, is animage with relatively high contrast and simple texture, where theforeground is connected to the lowermost border of the image. The secondrow of column 300 shows an image of a man positioned in the center as aforeground object and no part of the man extends to the borders of theimage. The third row shows an image with a starfish in the foregroundwhere the starfish has a block texture, part of which extends to theuppermost and right borders of the image. Finally, the forth row showsan image of woman with a complex repeated pattern on her blouse in theforeground which extends to the lowermost, left and right borders of theimage.

The gorilla image in column 300 contains a set of input markers,indicated by light shading 304, which the user inputs to identify atleast portions of the foreground of the image to be segmented. In thegorilla image, the gorilla is the foreground of the image to besegmented. Each of the other images in column 300 also contains a set ofuser markers on a foreground of the respective images. Note that none ofthe input markers in column 300 identify background regions. The imagesin column 302 show the output of the foreground image after the inputimage with foreground markers is processed according to an algorithmfollowing these teachings. The gorilla image in column 302 shows theforeground image 306 separate and distinct from any region that isclassified, considered or marked as background. The remaining space ofthe input image (column 300) that is not output as the foreground image(column 302) includes all regions that were classified, considered ormarked as background.

FIG. 3B shows another set of images in column 306 and column 308. Thefirst row in column 306 shows an image with a small girl in theforeground on the right and this is the only portion of that image thathas user marks 304 as foreground. The girl's dress has some texture andthe girl, including her dress, is connected to the right border of theimage. The next row shows an image of a tribal woman with a child on herback and both are marked by the user as foreground. The tribal woman'sskirt provides high texture and is attached to the lowermost border ofthe image while her headwrap extends to the uppermost border. It is ahigh contrast image with more color variation than some of the previousimages. The last row of column 310 shows an image of an airplane in theforeground extending to the border of the image on the right. Theconcrete in the background closely matches the color of the airplane andin the inventors' experimentation some prior art image segmentationtechniques had particular difficulty with distinguishing the cement fromthe airplane fuselage. This is a low contrast image.

Similar to FIG. 3A, the first column 306 in FIG. 3B shows a set of inputmarkers 304 on each image, indicated by light shading, identifying theforeground of the image to be segmented. Optional background markers 310are also used in the images of column 306. The set of input markers foreach image comprises both the markers indicating foreground 304 and themarkers indicating background 310. The images in column 308 show theoutput of the foreground image based on an exemplary embodiment of theinvention.

In FIG. 4 the top image shows a blown up version of the gorilla imagefrom FIG. 3A. The same input markers 304 are placed on the foreground ofthe image. The bottom image shows various different regions as used inembodiments of the invention. Regions 402 are the regions followingpre-segmentation that are defined by the set of input markers, whileborder regions 404 are possible foreground regions due to their colormatching with the user-marked foreground regions 402. The remainder 408of the lower portion of FIG. 4 is made up of regions that are determinedas background after intelligent background selection. Isolated regions406 are regions yet to be determined as foreground, background orambiguous regions.

FIG. 5 is a logic flow diagram that summarizes some of the aboveembodiments for segmenting an image using a computer, termed broadly assimply a device. Referring to FIG. 5, at block 502 a device using a setof user input foreground markers that identify a foreground of an imageto be segmented, defines N_(f) foreground regions and N_(b) backgroundregions, where N_(f) is an integer greater than zero and N_(b) is aninteger at least zero.

In one embodiment the set of user input consists of only the foregroundmarkers, and the N_(f) foreground regions and N_(b) background regionsare defined from the image using a mean shift algorithm. In anotherembodiment the set of user inputs includes user input foreground markersidentifying the foreground of the image to be segmented, and also userinput background markers identifying a background of the image to besegmented.

In general terms and as summarized at block 504, for each respectiveborder region that defines a border of the image and is not one of theN_(f) foreground regions, the above-described algorithm performs amulti-level assessment of the respective border region to determinewhether the respective border region is a possible foreground region.Each level of the multi-level assessment compares color similarity, withrespect to a threshold that is specific for that level, between therespective border region and a) any of the N_(f) foreground regions orb) of a region that satisfied the threshold of a previous level of themulti-level assessment.

In the specific embodiment detailed above, an initial level of themulti-level assessment comprises comparing a color distance between therespective border region and each of the N_(f) foreground regions andconcluding the respective border region is a similar border regionB_(sim) if the color distance is within a first threshold thr. While theimplementation above utilized the equation diffD_(i)=min(d(M_(f))−d(B_(i) ^(new) _(i=1,2, . . . ,m))) for this initiallevel, it is not necessary in all embodiments that the border regionsfirst be merged to yield the B_(i) ^(new) border regions beforeconducting this initial level assessment. For implementations in whichsuch border region merging is accomplished, then as detailed moreparticularly above prior to the initial level each border region B_(i)is merged with adjacent regions A_(j) thereof if an inter-color distancebetween the border region B_(i) and any respective adjacent region A_(j)is within a minimum color distance d_(i), to result in new borderregions B_(new) for which the color distance diff D_(i) is comparedagainst each of the N_(f) foreground regions during the initial level ofthe assessment.

For this multi-level assessment the specific implementation abovedetails that a subsequent level following the initial level of themulti-level assessment comprises comparing similarity of colorhistograms of each similar border regions B_(sim) that are output fromthe initial level as possible foreground regions against colorhistograms of other border regions that were not output from the initiallevel as possible foreground regions. It is these similar border regionsB_(sim) that satisfy the threshold of the previous level as step 504states. In this subsequent level, if the similarity of the colorhistograms is greater than a second threshold X then it is concludedthat the respective other border region B_(mat) is a possible foregroundregion. The possible foreground regions P_(f) resulting from both theinitial and subsequent levels of the assessment are then the similarborder regions B_(sim) and the other border regions B_(mat).

Returning to FIG. 5, the next main step at block 506 is that, from themulti-level assessment and the foreground markers, each region of theimage to be segmented is classified as one of a strong foregroundregion, a strong background region or an ambiguous region.

More specifically, the algorithm detailed by example above has thestrong foreground regions consisting of the N_(f) foreground regionsthat are identified by the user marked foreground markers and thepossible foreground regions P_(f). The strong background regions consistof the new border regions B_(new) that are not output from thesubsequent level as possible foreground regions B_(mat), along with anyfurther region identified by any user input background marker that mayhave been input, as a background of the image to be segmented. Adiscrimination filter is utilized to find the ambiguous regions G_(i)which are neither strong foreground regions nor strong backgroundregions. In the detailed implementation above the user-marked N_(f)foreground regions and the background are all input to thediscrimination filter which outputs the strong foreground regions,strong background regions and ambiguous regions. In this case thebackground is designated above as B_(mit), which is the user markedbackground regions M_(b) (if any) plus all the possible backgroundregions P_(b), where P_(b) is all the B_(new) that are not considered aspossible foreground regions P_(f).

FIG. 5 then has at block 508 classifying each of the respectiveambiguous regions as a weak foreground region or a weak backgroundregion. The more detailed implementation above classifies each of theserespective ambiguous regions G_(i) as a weak foreground region or a weakbackground region by comparing an inter-color distance between eachrespective ambiguous region G_(i) and its adjacent regions A_(i) andmerging the respective ambiguous region G_(i) with the adjacent regionA_(j) that is nearest in color distance to the respective ambiguousregion G_(i). This classifies each ambiguous region G_(i), after themerging, as either a weak foreground region W_(f) or a weak backgroundregion W_(b). More concisely, in this step there is a comparing of colorof each ambiguous region G_(i) against color of regions adjacent theretoand a merging the respective ambiguous region G_(i) with its adjacentregion having the most similar color.

FIG. 5 concludes with step 510 at which the strong foreground regionsare merged with the weak foreground regions and from this merging isoutput a foreground image of the (original) image to be segmented, andit is output separate and distinct from any region that is classified asone of the strong background regions or one of the weak backgroundregions. Such foreground images for output are shown by example atcolumn 302 of FIG. 3A, and also at column 308 of FIG. 3B. This output ofthe foreground image may be to a tangible computer readable memory, orto a graphical display such as a computer screen or projector.

This merging at step 510 is detailed as the pruning in the non-limitingexample above, which can be neatly summarized as a) merging the strongforeground regions with the weak foreground regions W_(f′), b) mergingthe strong background regions with the weak background regions W_(b); c)filling gaps that are surrounded by the merged strong and weakforeground regions; and d) depending on the user input foregroundmarkers, removing from the merged and filled strong and weak foregroundregions [part c) above] any isolated regions. From this moreparticularized merging/pruning step is output the foreground image,which in this case is separate and distinct from the merged strong andweak background regions and also separate and distinct from any removedisolated regions.

The logic diagram of FIG. 5 may be considered to illustrate theoperation of a method, and a result of execution of a computer programstored in a computer readable memory, and a specific manner in whichcomponents of an electronic device are configured to cause thatelectronic device to operate. The various blocks shown in FIG. 5 mayalso be considered as a plurality of coupled logic circuit elementsconstructed to carry out the associated function(s), or specific resultof strings of computer program code or instructions stored in a memory.

As shown in the high level block diagram of FIG. 6 a computing system isshown, which includes processing means such as at least one centralprocessing unit (CPU) 600, a graphics processing unit (GPU) 602, storingmeans such as at least one computer-readable memory (MEM) 604 storing atleast one computer program (PROG) 606, a display device 608, and aninput device 610, which are connected to communicate with one anotherthrough a bus 612. Also stored in the MEM 604 at reference number 606are the computer code or computer implementable algorithm(s) for thecomputer system to define foreground and background regions of an imageto be segmented, quantize similarity of border regions, and classifyregions as foreground or background according to the exemplaryembodiments described above.

The computer readable memory 604 may comprise a random access memory(RAM) and a mass storage. Various embodiments of the computer readableMEMs 604 include any data storage technology type which is suitable tothe local technical environment, including but not limited tosemiconductor based memory devices, magnetic memory devices and systems,optical memory devices and systems, fixed memory, removable memory, discmemory, flash memory, DRAM, SRAM, EEPROM and the like.

The CPU 600, the GPU 602, and the memory 604 are shown as separatecomponents, however in some embodiments, the memory 604 may beintegrated with one or both of the CPU 600 and the graphics processor602. In some embodiments, the memory 604 may be shared dynamically bythe CPU 600 and the graphics processor 602. In some embodiments, memory604 is used by the CPU 600, and a second graphics RAM is dedicated forthe use by the graphic processor 602. In some embodiments, the GPU 602may be integrated with the CPU 600. In some embodiments, the computersystem in FIG. 6 may lack a dedicated GPU 602, and the CPU 600 may beadapted to perform computations in place of the GPU 602. The CPU 600 andthe graphic processor 602 may be operating in either single precisionfloating point (32-bit) or double precision floating point (64-bit)format.

The input device 610 may be a commonly known device for providing userinput to a computer system, such as a mouse, a keyboard, a touch padand/or a touch screen, or a transducer (e.g., a microphone) inconjunction with voice recognition software. The display device 608 maybe a commonly known graphical display screen or projector for displayinggraphical images to a user. Although shown as separate blocks, thedisplay device 608 and the input device 610 may be integrated with oneanother, such as may be the case with a touch screen device. In someembodiments, the bus 612 may include a north bridge for connecting theCPU 600, the graphics processor 602, and the memory 604, and a southbridge for connecting the display device 608 and the input device 610.

In some embodiments, the input device 610 is adapted to provide commandsto select a set of input markers identifying a foreground and/or abackground of an image to the CPU 600. The commands may be enteredthrough a user interface displayed on the display device 608. Forexample, a user many select an image, select a portion of the imagewhich represents a background or foreground of the image through theuser interface. The image may be a picture, a graphic, a video or thelike. In some embodiments, the application of the input markers may betriggered by another program running on the computer system. The imagefile may be stored on the memory 604 and retrieved by the CPU 600 to bedisplayed on the display device 608.

In response to a command to select a set of input markers of an image,the CPU 600 may be adapted to execute codes stored in the memory 604. Insome embodiments the computer system may be adapted to apply embodimentsof the invention to a single image, a video comprising a series of stillimages, or some other video file such as a compressed video file. Theoutputted foreground image and/or regions classified, considered ormarked as background, may be displayed on the display device 608 andstored into the memory 604.

In general, the various embodiments of the computer system in FIG. 6 caninclude, but are not limited to workstations, servers, personal desktopcomputers, laptop or tablet computers, and even personal portabledigital devices having wireless communication capabilities, includingbut not limited to handheld or wearable computers such as cellularphones and smart phones, Google glass devices, and the like.

What is claimed is:
 1. A method of segmenting an image using a computer,the method comprising: using a set of user input foreground markersidentifying a foreground of an image to be segmented, defining N_(f)foreground regions and N_(b) background regions, where N_(f) is aninteger greater than zero and N_(b) is an integer at least zero; foreach respective border region that defines a border of the image and isnot one of the N_(f) foreground regions, performing a multi-levelassessment of the respective border region to determine whether therespective border region is a possible foreground region, where eachlevel of the multi-level assessment compares color similarity, withrespect to a threshold that is specific for that level, between therespective border region and any of the N_(f) foreground regions or of aregion that satisfied the threshold of a previous level of themulti-level assessment; from the multi-level assessment and theforeground markers, classifying each region of the image to be segmentedas one of a strong foreground region, a strong background region or anambiguous region; classifying each of the respective ambiguous regionsas a weak foreground region or a weak background region; and merging thestrong foreground regions with the weak foreground regions and outputtherefrom, to a tangible memory or to a graphical display, a foregroundimage of the image to be segmented, separate and distinct from anyregion classified as one of the strong background regions or weakbackground regions.
 2. The method according to claim 1, wherein the setof user inputs consists of only the foreground markers, and the N_(f)foreground regions and N_(b) background regions are defined from theimage using a mean shift algorithm.
 3. The method according to claim 1,wherein an initial level of the multi-level assessment comprises:comparing a color distance between the respective border region and eachof the N_(f) foreground regions and concluding the respective borderregion is a similar border region B_(sim) if the color distance iswithin a first threshold thr.
 4. The method according to claim 3,wherein prior to the initial level each border region B_(i) is mergedwith adjacent regions A_(j) thereof if an inter-color distance betweenthe border region B_(i) and any respective adjacent region A_(j) iswithin a minimum color distance d_(i), to result in new border regionsB_(new) for which the color distance diff D_(i) is compared against eachof the N_(f) foreground regions during the initial level of theassessment.
 5. The method according to claim 3, wherein a subsequentlevel following the initial level of the multi-level assessmentcomprises: comparing similarity of color histograms of each similarborder regions B_(sim) that are output from the initial level aspossible foreground regions against color histograms of other borderregions that were not output from the initial level as possibleforeground regions, and if the similarity of the color histograms isgreater than a second threshold X concluding that the respective otherborder region B_(mat) is a possible foreground region, such that thepossible foreground regions P_(f) are the similar border regions B_(sim)and the other border regions B_(mat).
 6. The method according to claim5, wherein: the strong foreground regions consist of the N_(f)foreground regions identified by the user marked foreground markers andthe possible foreground regions P_(f); the strong background regionsconsist of the new border regions B_(new) that are not output from thesubsequent level as possible foreground regions B_(mat) with any furtherregion identified by a user input background marker as a background ofthe image to be segmented; and the method further comprises utilizing adiscrimination filter to find the ambiguous regions G_(i) which areneither strong foreground regions nor strong background regions.
 7. Themethod according to claim 1, wherein classifying each of the respectiveambiguous regions G_(i) as a weak foreground region or a weak backgroundregion comprises: comparing color of each ambiguous region G_(i) againstcolor of regions adjacent thereto and merging the respective ambiguousregion G_(i) with its adjacent region having the most similar color. 8.The method according to claim 7, wherein merging the strong foregroundregions with the weak foreground regions comprises: merging the strongforeground regions with the weak foreground regions W_(f); merging thestrong background regions with the weak background regions W_(b);filling gaps that are surrounded by the merged strong and weakforeground regions; and depending on the user input foreground markers,removing from the merged and filled strong and weak foreground regionsany isolated regions; and outputting therefrom the foreground image,separate and distinct from the merged strong and weak background regionsand from any removed isolated regions.
 9. The method according to claim7, wherein each ambiguous region G_(i) is classified as either a weakforeground region W_(f) or a weak background region W_(b) by: formingset of adjacent regions S _(B)={A_(j)}_(j=1,2, . . . n), that areadjacent to the respective ambiguous region G_(i); calculate an intercolor distance between each respective ambiguous region G_(i) and eachof its adjacent regions S _(B); merging the respective ambiguous regionG_(i) with the adjacent region A_(j) for which the inter-color distanceis lowest; and classifying the respective ambiguous region G_(i) as aweak foreground region W_(f) or a weak background region W_(b) based onwhether the merged adjacent region A_(j) is a strong foreground regionS_(f) or a strong background region S_(b).
 10. A computer readablememory storing a program for segmenting an image, wherein the programcomprises computer executable code for causing a host device to: use aset of user input foreground markers identifying a foreground of animage to be segmented, to define N_(f) foreground regions and N_(b)background regions, where N_(f) is an integer greater than zero andN_(b) is an integer at least zero; for each respective border regionthat defines a border of the image and is not one of the N_(f)foreground regions, perform a multi-level assessment of the respectiveborder region to determine whether the respective border region is apossible foreground region, where each level of the multi-levelassessment compares color similarity, with respect to a threshold thatis specific for that level, between the respective border region and anyof the N_(f) foreground regions or of a region that satisfied thethreshold of a previous level of the multi-level assessment; from themulti-level assessment and the foreground markers, classify each regionof the image to be segmented as one of a strong foreground region, astrong background region or an ambiguous region; classify each of therespective ambiguous regions as a weak foreground region or a weakbackground region; and merge the strong foreground regions with the weakforeground regions and output therefrom, to a tangible memory or to agraphical display, a foreground image of the image to be segmented,separate and distinct from any region classified as one of the strongbackground regions or weak background regions.
 11. The computer readablememory according to claim 10, wherein the set of user inputs consists ofonly the foreground markers, and the N_(f) foreground regions and N_(b)background regions are defined from the image using a mean shiftalgorithm.
 12. The computer readable memory according to claim 10,wherein during an initial level of the multi-level assessment thecomputer executable code causes the host device to: compare a colordistance between the respective border region and each of the N_(f)foreground regions and concluding the respective border region is asimilar border region B_(sim) if the color distance is within a firstthreshold thr.
 13. The computer readable memory according to claim 12,wherein prior to the initial level the computer executable code causesthe host device to merge each border region B_(i) with adjacent regionsA_(j) thereof if an inter-color distance between the border region B_(i)and any respective adjacent region A_(j) is within a minimum colordistance d_(i), to result in new border regions B_(new) for which thecomputer executable code causes the host device to compare colordistance diff D_(i) against each of the N_(f) foreground regions duringthe initial level of the assessment.
 14. The computer readable memoryaccording to claim 12, wherein during a subsequent level following theinitial level of the multi-level assessment the computer executable codecauses the host device to: compare similarity of color histograms ofeach similar border regions B_(sim) that are output from the initiallevel as possible foreground regions against color histograms of otherborder regions that were not output from the initial level as possibleforeground regions, and if the similarity of the color histograms isgreater than a second threshold X to conclude that the respective otherborder region B_(mat) is a possible foreground region, such that thepossible foreground regions P_(f) are the similar border regions B_(sim)and the other border regions B_(mat).
 15. The computer readable memoryaccording to claim 14, wherein: the strong foreground regions consist ofthe N_(f) foreground regions identified by the user marked foregroundmarkers and the possible foreground regions P_(f); the strong backgroundregions consist of the new border regions B_(new) that are not outputfrom the subsequent level as possible foreground regions B_(mat) withany further region identified by a user input background marker as abackground of the image to be segmented; and the computer executablecode comprises a discrimination filter by which is found the ambiguousregions G_(i) which are neither strong foreground regions nor strongbackground regions.
 16. The computer readable memory according to claim10, wherein the computer executable code causes the host device toclassify each of the respective ambiguous regions G_(i) as a weakforeground region or a weak background region by: comparing aninter-color distance between each respective ambiguous region G_(i) andits adjacent regions A_(j) and merging the respective ambiguous regionG_(i) with the adjacent region A_(j) that is nearest in color distanceto the respective ambiguous region G_(i) such that each merged ambiguousregion G_(i) is classified as a weak foreground region W_(f) or a weakbackground region W_(b).
 17. The computer readable memory according toclaim 16, wherein merging the strong foreground regions with the weakforeground regions comprises: merging the strong foreground regions withthe weak foreground regions W_(f′); merging the strong backgroundregions with the weak background regions W_(b); filling gaps that aresurrounded by the merged strong and weak foreground regions; anddepending on the user input foreground markers, removing from the mergedand filled strong and weak foreground regions any isolated regions; andoutputting therefrom the foreground image, separate and distinct fromthe merged strong and weak background regions and from any removedisolated regions.
 18. The computer readable memory according to claim10, in combination with the host device comprising at least oneprocessor and the computer readable memory storing the program forsegmenting an image.
 19. A computer readable memory storing a programfor segmenting an image, wherein the program comprises computerexecutable code for causing a host device to: from inputs of an image tobe segmented and a set of user input foreground markers identifying aforeground of said image, determine strong foreground regions S_(f),strong background regions S_(b) and ambiguous regions G; classify eachambiguous region G_(i) as either a weak foreground region W_(f) or aweak background region W_(b) by comparing color of each ambiguous regionG_(i) against color of regions adjacent thereto and merging therespective ambiguous region G_(i) with its adjacent region having themost similar color; and merge the strong foreground regions S_(f) withthe weak foreground regions W_(f) and output therefrom, to a tangiblememory or to a graphical display, a foreground image of the image to besegmented, separate and distinct from any region classified as one ofthe strong background regions or weak background regions.
 20. Thecomputer readable memory according to claim 19, wherein each ambiguousregion G_(i) is classified as either a weak foreground region W_(f) or aweak background region W_(b) by: forming set of adjacent regions S_(B)={A_(j)}_(j=1,2, . . . n) that are adjacent to the respectiveambiguous region G_(i); calculate an inter color distance between eachrespective ambiguous region G_(i) and each of its adjacent regions S_(B); merging the respective ambiguous region G_(i) with the adjacentregion A_(j) for which the inter-color distance is lowest; andclassifying the respective ambiguous region G_(i) as a weak foregroundregion W_(f) or a weak background region W_(b) based on whether themerged adjacent region A_(j) is a strong foreground region S_(f) or astrong background region S_(b).